HADOOP-19622: [ABFS][ReadAheadV2] Implement Read Buffer Manager V2 with improved aggressiveness #7832

anujmodi2021 · 2025-07-29T07:44:46Z

Description of PR

JIRA: https://issues.apache.org/jira/browse/HADOOP-19622

Implementing ReadBufferManagerV2 as per the new design document.
Following capabilities are added to ReadBufferManager:

Configurable minimum and maximum number of prefetch threads.
Configurable minimum and maximum size of cached buffer pool
Dynamically adjusting thread pool size and buffer pool size based on workload requirement and resource utilization but within the limits defined by user.
Mapping prefetched data to file ETag so that multiple streams reading same file can share the cache and save TPS.

For more details on design doc please refer to the design doc attached to parent JIRA: https://issues.apache.org/jira/browse/HADOOP-19596

How was this patch tested?

TBA

… into RBMV2_HADOOP-19622

anmolanmol1234 · 2025-10-29T11:27:24Z

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java

        LOG.debug("issuing read ahead requestedOffset = {} requested size {}",
            nextOffset, nextSize);
-        readBufferManager.queueReadAhead(this, nextOffset, (int) nextSize,
+        getReadBufferManager().queueReadAhead(this, nextOffset, (int) nextSize,


Needs changes in constructor to return an instance of readbuffermanager v2 if enabled

… into RBMV2_HADOOP-19622

bhattmanish98

+1 LGTM

anujmodi2021 · 2025-10-30T11:33:00Z

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 874, Failures: 0, Errors: 0, Skipped: 214
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 877, Failures: 0, Errors: 0, Skipped: 166
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey-DFS

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 716, Failures: 0, Errors: 0, Skipped: 279
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 874, Failures: 0, Errors: 0, Skipped: 225
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 723, Failures: 0, Errors: 0, Skipped: 137
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-DFS

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 713, Failures: 0, Errors: 0, Skipped: 281
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 720, Failures: 0, Errors: 0, Skipped: 149
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 715, Failures: 0, Errors: 0, Skipped: 195
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 748, Failures: 0, Errors: 0, Skipped: 223
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-OAuth-DFS-IngressBlob

[WARNING] Tests run: 202, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 713, Failures: 0, Errors: 0, Skipped: 278
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

hadoop-yetus · 2025-10-30T12:15:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 10 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 24s		trunk passed
+1 💚	compile	0m 25s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 23s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	0m 18s		trunk passed
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	javadoc	0m 23s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	spotbugs	0m 45s	/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html	hadoop-tools/hadoop-azure in trunk has 178 extant spotbugs warnings.
+1 💚	shadedclient	14m 11s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	14m 23s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 20s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 18s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 20s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 11s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 1 new + 5 unchanged - 9 fixed = 6 total (was 14)
+1 💚	mvnsite	0m 21s		the patch passed
-1 ❌	javadoc	0m 17s	/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 54 new + 1464 unchanged - 0 fixed = 1518 total (was 1464)
-1 ❌	javadoc	0m 16s	/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 22 new + 1390 unchanged - 0 fixed = 1412 total (was 1390)
-1 ❌	spotbugs	0m 46s	/new-spotbugs-hadoop-tools_hadoop-azure.html	hadoop-tools/hadoop-azure generated 9 new + 168 unchanged - 10 fixed = 177 total (was 178)
+1 💚	shadedclient	14m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 17s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 21s		The patch does not generate ASF License warnings.
		59m 45s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-azure
	Unknown bug pattern AT_NONATOMIC_64BIT_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setOffset(long) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setOffset(long) At ReadBuffer.java:[line 91]
	Unknown bug pattern AT_NONATOMIC_64BIT_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setTimeStamp(long) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setTimeStamp(long) At ReadBuffer.java:[line 172]
	Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setAnyByteConsumed(boolean) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setAnyByteConsumed(boolean) At ReadBuffer.java:[line 196]
	Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setBufferindex(int) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setBufferindex(int) At ReadBuffer.java:[line 123]
	Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setFirstByteConsumed(boolean) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setFirstByteConsumed(boolean) At ReadBuffer.java:[line 180]
	Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setLastByteConsumed(boolean) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setLastByteConsumed(boolean) At ReadBuffer.java:[line 188]
	Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setLength(int) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setLength(int) At ReadBuffer.java:[line 99]
	Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setRequestedLength(int) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setRequestedLength(int) At ReadBuffer.java:[line 107]
	Unknown bug pattern AT_STALE_THREAD_WRITE_OF_PRIMITIVE in org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setStatus(ReadBufferStatus) At ReadBuffer.java:org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setStatus(ReadBufferStatus) At ReadBuffer.java:[line 141]

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/30/artifact/out/Dockerfile
GITHUB PR	#7832
JIRA Issue	HADOOP-19622
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c80b7679a995 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c7f986e`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/30/testReport/
Max. process+thread count	636 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/30/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021 · 2025-10-31T06:50:06Z

Thanks everyone for reviewing this.

Spotbugs and Javadocs warnings are due to https://issues.apache.org/jira/browse/HADOOP-19731
Rest all looks good. Going ahead with the merge

…th improved aggressiveness (apache#7832) Contributed by Anuj Modi

…th improved aggressiveness (#7832) (#8175) Contributed by Anuj Modi

anujmodi2021 added 2 commits July 28, 2025 21:44

RBMV2 Implementation

37898b4

Test Improvements

28cb97f

github-actions bot added trunk TOOLS ABFS labels Jul 29, 2025